Using the ISO/IEC 9126 product quality model to classify defects : a Controlled Experiment

Background: Existing software defect classification schemes support multiple tasks, such as root cause analysis and process improvement guidance. However, existing schemes do not assist in assigning defects to a broad range of high level software goals, such as software quality characteristics like functionality, maintainability, and usability. Aim: We investigate whether a classification based on the ISO/IEC 9126 software product quality model is reliable and useful to link defects to quality aspects impacted. Method: Six different subjects, divided in two groups with respect to their expertise, classified 78 defects from an industrial web application using the ISO/IEC 9126 quality main characteristics and sub-characteristics, and a set of proposed extended guidelines. Results: The ISO/IEC 9126 model is reasonably reliable when used to classify defects, even using incomplete defect reports. Reliability and variability is better for the six high level main characteristics of the model than for the 22 sub- characteristics. Conclusions: The ISO/IEC 9126 software quality model provides a solid foundation for defect classification. We also recommend, based on the follow up qualitative analysis performed, to use more complete defect reports and tailor the quality model to the context of us

Are Delayed Issues Harder to Resolve? Revisiting Cost-to-Fix of Defects throughout the Lifecycle

Many practitioners and academics believe in a delayed issue effect (DIE); i.e. the longer an issue lingers in the system, the more effort it requires to resolve. This belief is often used to justify major investments in new development processes that promise to retire more issues sooner. This paper tests for the delayed issue effect in 171 software projects conducted around the world in the period from 2006--2014. To the best of our knowledge, this is the largest study yet published on this effect. We found no evidence for the delayed issue effect; i.e. the effort to resolve issues in a later phase was not consistently or substantially greater than when issues were resolved soon after their introduction. This paper documents the above study and explores reasons for this mismatch between this common rule of thumb and empirical data. In summary, DIE is not some constant across all projects. Rather, DIE might be an historical relic that occurs intermittently only in certain kinds of projects. This is a significant result since it predicts that new development processes that promise to faster retire more issues will not have a guaranteed return on investment (depending on the context where applied), and that a long-held truth in software engineering should not be considered a global truism.Comment: 31 pages. Accepted with minor revisions to Journal of Empirical Software Engineering. Keywords: software economics, phase delay, cost to fi

Software Engineering Research/Developer Collaborations (C104)

The goal of this collaboration was to produce Flight Software Branch (FSB) process standards for software inspections which could be used across three new missions within the FSB. The standard was developed by Dr. Forrest Shull (Fraunhofer Center for Experimental Software Engineering, Maryland) using the Perspective-Based Inspection approach, (PBI research has been funded by SARP) , then tested on a pilot Branch project. Because the short time scale of the collaboration ruled out a quantitative evaluation, it would be decided whether the standard was suitable for roll-out to other Branch projects based on a qualitative measure: whether the standard received high ratings from Branch personnel as to usability and overall satisfaction. The project used for piloting the Perspective-Based Inspection approach was a multi-mission framework designed for reuse. This was a good choice because key representatives from the three new missions would be involved in the inspections. The perspective-based approach was applied to produce inspection procedures tailored for the specific quality needs of the branch. The technical information to do so was largely drawn through a series of interviews with Branch personnel. The framework team used the procedures to review requirements. The inspections were useful for indicating that a restructuring of the requirements document was needed, which led to changes in the development project plan. The standard was sent out to other Branch personnel for review. Branch personnel were very positive. However, important changes were identified because the perspective of Attitude Control System (ACS) developers had not been adequately represented, a result of the specific personnel interviewed. The net result is that with some further work to incorporate the ACS perspective, and in synchrony with the roll out of independent Branch standards, the PBI approach will be implemented in the FSB. Also, the project intends to continue its collaboration with the technology provider (Dr. Forrest Shull) past the end of the grant, to allow a more rigorous quantitative evaluation

"GSFC FSB Application of Perspective-Based Inspections"

The scope of work described in our proposal consisted of developing inspection standards targeted to Branch-specific types of defects (gained from analysis of Branch project defect histories), and including Branch-relevant perspectives and questions to guide defect detection. The tailored inspection guidelines were to be applied on real Branch projects with support as needed from the technology infusion team. This still accurately describes the scope of work performed. It was originally proposed that the Perspective-Based inspection standard would be applied on three projects within the Branch: GPM, JWST, and SDO. Rather than apply the proposed standard to all three, we inserted a new step, in which the standard was instead applied on a single pilot project, cFE (described above). This decision was a good match for the Branch goals since, due to the "design for reuse" nature of cFE, inspections played an even more crucial than usual role in that development process. Also, since cFE is being designed to provide general-purpose functionality, key representatives fiom our target projects were involved in inspections of cFE to provide perspectives from different missions. In this way, they could get some exposure to and the chance to provide feedback on the proposed standards before applying them on their own projects. The Branch-baselined standards will still be applied on GPM, JWST, and SDO, although outside the time frame of this funding. Finally, we originally proposed using the analysis of Branch defect sources to indicate in which phases Perspective-Based inspections could provide the best potential for future improvement, although experience on previous Branch projects suggested that our efforts would likely be focused on requirements and code inspections. In the actual work, we focused exclusively on requirements inspections, as this was the highest-priority work currently being done on our cFE pilot project

A Reference Model for Software and System Inspections. White Paper

Software Quality Assurance (SQA) is an important component of the software development process. SQA processes provide assurance that the software products and processes in the project life cycle conform to their specified requirements by planning, enacting, and performing a set of activities to provide adequate confidence that quality is being built into the software. Typical techniques include: (1) Testing (2) Simulation (3) Model checking (4) Symbolic execution (5) Management reviews (6) Technical reviews (7) Inspections (8) Walk-throughs (9) Audits (10) Analysis (complexity analysis, control flow analysis, algorithmic analysis) (11) Formal method Our work over the last few years has resulted in substantial knowledge about SQA techniques, especially the areas of technical reviews and inspections. But can we apply the same QA techniques to the system development process? If yes, what kind of tailoring do we need before applying them in the system engineering context? If not, what types of QA techniques are actually used at system level? And, is there any room for improvement.) After a brief examination of the system engineering literature (especially focused on NASA and DoD guidance) we found that: (1) System and software development process interact with each other at different phases through development life cycle (2) Reviews are emphasized in both system and software development. (Figl.3). For some reviews (e.g. SRR, PDR, CDR), there are both system versions and software versions. (3) Analysis techniques are emphasized (e.g. Fault Tree Analysis, Preliminary Hazard Analysis) and some details are given about how to apply them. (4) Reviews are expected to use the outputs of the analysis techniques. In other words, these particular analyses are usually conducted in preparation for (before) reviews. The goal of our work is to explore the interaction between the Quality Assurance (QA) techniques at the system level and the software level

Exploring the Department of Defense Software Factbook

The Carnegie Mellon Software Engineering Institute (SEI) conducted an analysis of software engineering data owned and maintained by the Department of Defense (DoD) to produce high-level, DoD-wide heuristics and domain-specific benchmark data. This work yielded basic facts about software projects, such as averages, ranges, and heuristics for requirements, size, effort, and duration. Factual, quantitatively-derived statements were reported to provide users with easily digestible benchmarks. Findings were also presented by system type, or super domain. The analysis in this area focused on identifying the most and least expensive projects and the best and worst projects within three super domains: real time, engineering, and automated information systems. It also provided insight into the differences between system domains and contained domain-specific heuristics. Finally, correlations were explored among requirements, size, duration, and effort and the strongest models for predicting change were described. The goal of this work was to determine how well the data could be used to answer common questions related to planning or replanning software projects. The paper provides a high-level overview of the SEI's research and primary findings.Naval Postgraduate School Acquisition Research Progra

Organizing the Technical Debt Landscape

To date, several methods and tools for detecting source code and design anomalies have been developed. While each method focuses on identifying certain classes of source code anomalies that potentially relate to technical debt (TD), the overlaps and gaps among these classes and TD have not been rigorously demonstrated. We propose to construct a seminal technical debt landscape as a way to visualize and organize research on the subjec

Defect Categorization: Making Use of a Decade of Widely Varying Historical Data

This paper describes our experience in aggregating a number of historical datasets containing inspection defect data using different categorizing schemes. Our goal was to make use of the historical data by creating models to guide future development projects. We describe our approach to reconciling the different choices used in the historical datasets to categorize defects, and the challenges we faced. We also present a set of recommendations for others involved in classifying defects

Understanding automated and human-based technical debt identification approaches-a two-phase study

Context: The technical debt (TD) concept inspires the development of useful methods and tools that support TD identification and management. However, there is a lack of evidence on how different TD identification tools could be complementary and, also, how human-based identification compares with them. Objective: To understand how to effectively elicit TD from humans, to investigate several types of tools for TD identification, and to understand the developers’ point of view about TD indicators and items reported by tools. Method: We asked developers to identify TD items from a real software project. We also collected the output of three tools to automatically identify TD and compared the results in terms of their locations in the source code. Then, we collected developers’ opinions on the identification process through a focus group. Results: Aggregation seems to be an appropriate way to combine TD reported by developers. The tools used cannot help in identifying many important TD types, so involving humans is necessary. Developers reported that the tools would help them to identify TD faster or more accurately and that project priorities and current development activities are important to be considered together, along with the values of principal and interest, when deciding to pay off a debt. Conclusion: This work contributes to the TD landscape, which depicts an understanding between different TD types and how they are best discovered

Full Lifecycle Defect Management

This viewgraph presentation describes a defect management scheme for projects at various NASA centers